1. Field of the Invention
The present invention relates to concealed regions complementing system of free viewpoint video images, and more particularly, to concealed regions complementing system of free viewpoint video images in which when the video images from free viewpoints are generated, they are complemented using a background buffer with respect to the regions where corresponding pixels can not be obtained due to concealment.
2. Description of the Related Art
Free viewpoint video images gain the spotlight as one of materials of next-generation interactive video image applications. The free viewpoint video images are images which are obtained if a user freely selects viewpoints with respect to a subject. Since there exist infinite viewpoints to be selected by the user, it is impractical to prepare all video images corresponding to the viewpoints.
Thereupon, it is conventionally used a technique to render images at the free viewpoints using a material of a subject described by 3-D information. For example, the following non-patent document 1 proposes a technique which uses 3-D position information as a material, and the following non-patent document 2 proposes a technique which uses the video images of a subject shoot from many directions as materials.
However, these techniques have a problem that a special apparatus is required for obtaining precise 3-D information, and since it is necessary to fix a large number of cameras for shooting a subject from various directions, shooting environment is limited.
The development of transfer of additional information together with video images has been pursued for the sake of providing interactive video image application between itself and a user. For example, if depth information or parallax information is transferred as the additional information, it becomes possible to generate video images from free viewpoints (free viewpoint video images, hereinafter).
With this tendency, various proposals for generating the free viewpoint video images using the depth information or the parallax information have been made. For example, the following non-patent document 3 proposes a corresponding technique for precisely generating the depth information, and this makes it possible to precisely calculate the depth information. The following non-patent document 4 proposes a technique for rendering each object precisely from another viewpoint.
When the free viewpoint video images are produced using these techniques, since the background regions are not taken into account, there is a problem that concealed regions are not sufficiently rendered.
There is a technique for estimating 3-D information based on a reference video image and the depth information (depth map) of each pixel of the reference video image.
In the case of the technique for estimating the 3-D information based on the reference video image and the depth map of each pixel of the reference video image, a range of viewpoints which can be selected is limited, but since the depth map can be handled as the associated information of the reference video image, this technique has excellent consistency with respect to encoded format which corresponds to transfer of the associated information such as MPEG-4, this technique can be used also in an environment such as outdoor where a large number of cameras can not be fixed, and this technique has a merit that versatility is high. However, in this technique, since pixel information of the background regions which are concealed by the foreground regions are missing, there is a problem that pixels which can not be rendered are generated in the free viewpoint video images.
The following non-patent document 5 proposes a generating method of free viewpoint video images in which a depth map of each video image is produced based on a plurality of video images. According to this method, since a plurality of video images and the depth maps corresponding to the video images are required, there is a problem that the amount of information to be transferred is increased.
The following non-patent document 6 proposes a method in which a foreground video image and a background video image are separated from each other, and the background video image is complemented using a sprite. According to this method, however, since it is absolutely necessary that the background video image is known, there is a problem that if the background video image can not be obtained previously, it is difficult to complement.
[Non-patent document 1] Saied Moezzi, Li-Cheng Tai, and Philippe Gerard “Vertual View Generation for 3-D Digital Video”, IEEE Multimedia, Vol. 4, No. 1, pp 18 to 26, 1997
[Non-patent document 2] Naho Hashimoto, Hideo Saito, intermediate picture generation from multi-viewpoint picture in soccer scene, Denshi Jouhou Tsushin Gakkai Gijyutu Houkoku, PRMU2001-151, November 2001, pp. 87 to 94
[Non-patent document 3] “Actual time stereo matching system using high precision repeating gradient for multi-viewpoint transfer” of Kawada et al, 2002, picture media gakkai Touki Taikai, No. 7 to 5, November 2002
[Non-patent document 4] “Viewpoint following type stereo picture rendering method by complement and re-construction of multi-viewpoint video image”, &J79-D-II, No. 5, pp 803 to 811, May 1996
[Non-patent document 5] “Virtual viewpoint video image production using a plurality of depth maps” by Hiruma et al, Denshi Jouhou Tuushin Gakkai Ronnbun, Vol. J84-D-II, No. 5, pp. 805 to 811, May 2001
[Non-patent document 6] “Global motion calculating method and application of encoding for producing sprite”, Denshi Jouhou Tuushin Gakkai Ronnbun, Vol. J84-D-II, No 2, pp. 535-544, February 2002